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Human behavior representation (HBR) is an elusive, yet critical goal for many in the 
simulation community. Requirement specifications related to HBR often exceed current 
capabilities. There exist a number of tools, techniques and frameworks to model and 
simulate HBR, but they are constrained and do not generalize well. Even with a vibrant 
research community, certain HBR characteristics remain beyond our grasp, unless some 
unforeseen disruptive technologies emerge. We survey the state of the prac ... 

8 Interactive information r etrieval systems: from u ser centered interface design to 

sof^Ar© design 
P. Mulhem, L. Nigay 

August 1996 Proceedings of the 19th annual international ACM SIGZR conference on 
Research and development in information retrieval SIGIR '96 

Publisher: ACM Press 

Full text available: ^pdf d.48 M B) Additional Information; full citation, refe ren ces, citings, index terms 




C onfirmation in multimodal systems 

David R. McGee, Philip R. Cohen, Sharon Oviatt 

August 1998 Proceedings of the 36th annual meeting on Association for 

Computational Linguistics - Volume 2 , Proceedings of the 17th 
international conference on Computational linguistics - Volume 2 

Publisher: Association for Computational Linguistics 

Full text available:^ pdf( 1.1 2 MB ) Additional Information: fu ll citat io n , abstract, r eference s, citin gs 

Systems that attempt to understand natural human input make mistakes, even humans. 
However, humans avoid misunderstandings by confirming doubtful input. Multimodal 
systems- —those that combine simultaneous input from more than one modality, for 
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gesture and typing for communicating with a computer. This paper presents an analysis of 
the integration of multiple communication modalities within an interactive system. To do 
so, a software engineering perspective is adopted. First, the notion of "multimodal 
system" is clarified. We aim at proving that two main features of a multimodal system are 
the concurrency of processing and the fusion ... 

Keywords: concurrency, data fusion, design space, modality, multimodal interaction, 
software architecture, taxonomy 
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L Ardissono, A. Goy, G. Petrone 

July 2003 Proceedings of the second international joint conference on Autonomous 
agents and multiagent systems AAMAS '03 

Publisher: ACM Press 

Full text available* f 5 ") odf(228 62 KB) Additional Information: full citatio n, abstract , references , citings, index 
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The emerging standards for the publication of Web Services enable the invocation of 
services having simple interaction protocols, but they fail to support complex e-business 
interactions, where the peers exchange several messages. In order to. extend the classes 
of services which can be invoked by the consumers, we propose a conversational model 
supporting the management of complex interactions between clients and Web Services. 
Our model supports the consumer in the management of a conversation w ... 

Keywords: conversational agents, web services 



13 Interactiv e knowled g e capture I: Knowled g e mana ge ment for product maturit y 
Guy A. Boy 

October 2005 Proceedings of the 3rd international conference on Knowledge capture 
K-CAP '05 

Publisher: ACM Press 

Full text available:^ pdf(2 23. 26 KB ) Additional Information: full citation , abstract , references , index terms 

When a new product is delivered, it seldom meets all customer needs. The mature phase 
of a product is driven by customer needs. It requires a human-centered development 
cycle. As a result, the company should be able to listen the voice of its customers. Most 
industrial companies are driven by engineers and by technology itself. If current 
technology is to serve all actors of the life cycle of a product, related companies need to 
change their ways of dealing with maturity. They have to stop being ... 

Keywords: active design documents, design and development, knowledge management, 
maturity, user experience 
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This document summarizes contributions and discussions from two workshops that took 
place in November 2000 and July 2001. It presents some visions of NLP-related 
applications that may become reality within ten years from now. It investigates the 
technological requirements that must be met in order to make these visions realistic and 
sketches milestones that may help to measure our progress towards these goals. 

15 Establishing an d maintaining long-t erm hu man-computer relationships 
Timothy W. Bickmore, Rosalind W. Picard 

June 2005 ACM Transactions on Computer-Human Interaction (TOCHI), volume 12 issue 
2 

Publisher: ACM Press 

Full text available* f*) pdf(1 68 MB) Additional Information: full citation , abstract , references , citings, index 
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This research investigates the meaning of "human-computer relationship" and. presents 
techniques for constructing, maintaining, and evaluating such relationships, based on 
research in social psychology, sociolinguistics, communication and other social sciences. 
Contexts in which relationships are particularly important are described, together with 
specific benefits (like trust) and task outcomes (like improved learning) known to be 
associated with relationship quality. We especially c ... 

Keywords: Human-computer interaction, embodied conversational agent, relational 
agent, social interface 



16 Req uirements interaction mana g ement 

William N. Robinson, Suzanne D. Pawlowski, Vecheslav Volkov 
June 2003 ACM Computing Surveys (CSUR), volume 35 issue 2 
Publisher: ACM Press 

Full text available- 15| pdf(1 24 MB) Additional Information: full citation , abstract , references , cjtings, index 
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Requirements interaction management (RIM) is the set of activities directed toward the 
discovery, management, and disposition of critical relationships among sets of 
requirements, which has become a critical area of requirements engineering. This survey 
looks at the evolution of supporting concepts and their related literature, presents an 
issues-based framework for reviewing processes and products, and applies the framework 
in a review of RIM state-of-the-art. Finally, it presents seven researc ... 

Keywords: KAOS, KATE, Oz, Requirements engineering, Telos, WinWin, analysis and 
design, composite system, deficiency driven design, dependency analysis, distributed 
intentionality, interaction analysis, software cost reduction (SCR)., system architecture, 
system specification, viewpoints 
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Publisher: ACM Press 

Full text available: ^pdf( 281.80 KB ) Additional Information: full citation , a bstract , references , index terms 

Within the framework of the project NICE (Natural Interactive Communication for 
Edutainment) [2], we have been developing an educational and entertaining computer 
game that allows children and teenagers to interact with a conversational character 
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impersonating the fairy tale writer H.C. Andersen (HCA). The rationale behind our system 
is to make kids learn about HCA's life, fairy tales and historical period while playing and 
having fun. We report on the character's generation and realization of b ... 

Keywords: edutainment, embodied conversational agent, multimodal output, user 
interface 



18 S ALT: a n X ML a pplic ation for web-based multimodal dialo g management Q 
* Kuansan Wang 

September 2002 Proceedings of the 2nd workshop on NLP and XML - Volume 17 
NLPXML '02 
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Full text available: ^.Bdf( 186,90 KB). Additional Information: full djtaJion , abstract, relereoces 

This paper describes the Speech Application Language Tags, or SALT, an XML based 
spoken dialog standard for multimodal or speech-only applications. A key premise in SALT 
design is that speech-enabled user interface shares a lot of the design principles and 
computational requirements with the graphical user interface (GUI). As a result, it is 
logical to introduce into speech the object-oriented, event-driven model that is known to 
be flexible and powerful enough in meeting the requirements for re ... 

19 A BDI a g ent architecture for dialo g ue modellin g and coordination in a smart personal Q 
assistant 

Wayne Wobcke, Van Ho, Anh Nguyen, Alfred Krzywicki 

April 2006 Proceedings of the 2005 NICTA-HCSNet Multimodal User Interaction 
Workshop - Volume 57 MMUI '05 

Publisher: Australian Computer Society, Inc. 

Full text available: pdf( 160.03 KB ) Additional Information: full cita tion , abstract, refere nces, index term s 

In this paper, we discuss the architectural aspects of a Smart Personal Assistant (SPA) 
system that enables users to access a range of applications from a range of devices using 
multi-modal natural language dialogue. Each back-end application is a personal assistant 
specializing in one specific task such as e-mail or calendar management, and typically 
each has its own user model, enabling it to adapt to the user's changing preferences. The 
PDA interface to the SPA must present the system as a si ... 

20 Extractin g usability information from user interface events Q 
David M. Hilbert, David F. Redmiles 

December 2000 ACM Computing Surveys (CSUR), volume 32 issue 4 
Publisher: ACM Press 

Full text available: IB pdf(1 .50 MB) Additional Information: fuJLcjtation, abstract, references, citings, index 
IAj terms, review 

Modern window-based user interface systems generate user interface events as natural 
products of their normal operation. Because such events can be automatically captured 
and because they indicate user behavior with respect to an application's user interface, 
they have long been regarded as a potentially fruitful source of information regarding 
application usage and usability. However, because user interface events are typically 
voluminos and rich in detail, automated support is generally ... 

Keywords: human-computer interaction, sequential data analysis, usability testing, user 
interface event monitoring 
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This research investigates the meaning of "human-computer relationship" and presents 
techniques for constructing, maintaining, and evaluating such relationships, based on 
research in social psychology, sociolinguistics, communication and other social sciences. 
Contexts in which relationships are particularly important are described, together with 
specific benefits (like trust) and task outcomes (like improved learning) known to be 
associated with relationship quality. We especially c ... 
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Within the framework of the project NICE (Natural Interactive Communication for 
Edutainment) [2], we have been developing an educational and entertaining computer 
game that allows children and teenagers to interact with a conversational character 
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As household appliances grow in complexity and sophistication, they become harder and 
harder to use, particularly because of their tiny display screens and limited keyboards. 
This paper describes a strategy for building natural language interfaces to appliances that 
circumvents these problems. Our approach leverages decades of research on planning 
and natural language interfaces to databases by reducing the appliance problem to the 
database problem; the reduction provably maintains desirable pro ... 

Keywords: appliance, database, natural language interface, planner 
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ISIS (Intelligent Speech for Information Systems) is a trilingual spoken dialog system 
(SDS) for the stocks domain. It handles two dialects of Chinese (Cantonese and 
Putonghua) as well as English— the predominant languages in our region. The system 
supports spoken language queries regarding stock market information and simulated 
personal portfolios. The conversational interface is augmented with a screen display that 
can capture mouse-clicks as well as textual input by typing or stylus-writing. ... 
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Human behavior representation (HBR), is an elusive, yet critical goal for many in the 
simulation community. Requirement specifications related to HBR often exceed current 
capabilities. There exist a number of tools, techniques and frameworks to model and 
simulate HBR, but they are constrained and do not generalize well. Even with a vibrant 
research community, certain HBR characteristics remain beyond our grasp, unless some 
unforeseen disruptive technologies emerge. We survey the state of the prac ... 
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Online purchase from e-stores is getting popular as the prevalence of electronic 
commerce. At current stage, most e-stores resemble vending machines rather than real 
stores because they lack clerks to persuade prospects into buying products and to bargain 
with the customers for making a good deal. This research designs an easy-to-use and 
autonomous sales-agent to act as a virtual clerk in an e-store, and then investigates 
whether an e-store with thjs virtual clerk could increase customers' produ ... 
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We are reaching a crisis with-design of user interfaces for consumer electronics. Flashing 
12:00 time indicators, push-and-hold buttons, and interminable modes and menus are all 
symptoms of trying to maintain a one-to-one correspondence between functions and 
physical controls, which becomes hopeless as the number of capabilities of devices grows. 
We propose instead to orient interfaces around the goals that users have for the use of 
devices. We present Roadie, a user interface agent that provides ... 

Keywords: commonsense reasoning, consumer electronics, goal-oriented interfaces, 
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We describe an implemented architecture for programming the responses of collaborative 
interface agents out of easily composable and reusable plug-in components, and discuss 
the underlying theoretical and practical issues. The power of the architecture comes 
primarily from a rich representation of collaborative discourse state, which includes a 
focus stack and plan tree. The architecture also provides a useful separation between the 
principles and preferences underlying an agent's behavior. We i ... 
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An implemented system is described which allows a software agent to collaborate with a 
human user using a shared application window. The system automatically controls input 
permission and also provides mechanisms for signalling and communication. A 
generalization of the system to multiple users and agents, called NShare, is compared 
with common window-sharing tools, such as SharedX. This work is part of a larger agenda 
to apply principles of human collaboration and discourse structure to human-c ... 
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Full text available:^ pdf( 1. 12 MB) Additional Information: full citation , abstract , references , citings 

Systems that attempt to understand natural human input make mistakes, even humans. 
However, humans avoid misunderstandings by confirming doubtful input. Multimodal 
systems— those that combine simultaneous input from more than one modality, for 
example speech and gesture-have historically been designed so that they either request 
confirmation of speech, their primary modality, or not at all. Instead, we experimented 
with delaying confirmation until after the speech and gesture were combin ... 

Keywords: confirmation, disambiguation, multimodal, uncertainty 
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Most computational spoken dialogue systems take a "literary" approach to reference 
resolution. With this type of approach, entities that are mentioned by a human interactor 
are unified with elements in the world state based on the same principles that guide the 
process during text interpretation. In human-to-human interaction, however, referring is 
a much more collaborative process. Participants often under-specify their referents, 
relying on their discourse partners for feedback if more informa ... 

Keywords: HQ, dialogue systems, eye tracking, reference resolution 
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. [A) : terms 

Multimodal interaction enables the user to employ different modalities such as voice, 
gesture and typing for communicating with a computer. This paper presents an analysis of 
the integration of multiple communication modalities within an interactive system. To do 
so, a software engineering perspective is adopted. First, the notion of "multimodal 
system" is clarified. We aim at proving that two main features of a multimodal system are 
the concurrency of processing and the fusion ... 

Keywords: concurrency, data fusion, design space, modality, multimodal interaction, 
software architecture, taxonomy 
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Establishing and maintainin g lon g -term human -computer relationships 
Timothy W. Bickmore, Rosalind W. Picard 

June 2005 ACM Transactions on Computer-Human Interaction (TOCHI), volume 12 issue 



Publisher: ACM Press 

Full text available: f£| pdf( 1.68 MB ) 



Additional Information: full cit ation , abstra ct, reference s, citings, index, 
terms 



This research investigates the meaning of "human-computer relationship" and presents 
techniques for constructing, maintaining, and evaluating such relationships, based on 
research in social psychology, sociolinguistics, communication and other social sciences. 
Contexts in which relationships are particularly important are described, together with 
specific benefits (like trust) and task outcomes (like improved learning) known to be 
associated with relationship quality. We especially c ... 

Keywords: Human-computer interaction, embodied conversational agent, relational 
. agent, social interface 



2 Computing . currj^la_2001 

,£jv September 2001 Journal on Educational Resources in Computing (JERIC) 
^ Publisher: ACM Press 



Full text available: g|pdff 61 3.63 KB ) 
g| html (2.78 KB) 



Additional Information: f ull citatio n, r eferences , citings, index ter ms 



3 Long pa pers: re com m endation a nd instru ction: A nimating an inte ractive 

conversational character for an educational g ame s ystem 
^ Andrea Corradini, Manish Mehta, Niels-Ole Bernsen, Marcela Charfuelan 

January 2005 Proceedings of the 10th international conference on Intelligent user 
interfaces IUI '05 

Publisher: ACM Press 

Full text available: ^|pdf (281.80 KB) Additional Information: full citation , abstract , references , index terms 

Within the framework of the project NICE (Natural Interactive Communication for 
Edutainment) [2], we have been developing an educational and entertaining computer 
game that allows children and teenagers to interact with a conversational character 
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impersonating the fairy tale writer H.C. Andersen (HCA). The rationale behind our system 
is to make kids learn about HCA's life, fairy tales and historical period while playing and 
having fun. We report on the character's generation and realization of b ... 

Keywords: edutainment, embodied conversational agent, multimodal output, user 
interface 
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Full Technical Pa pers: A re li ab l e natural l anguage interface to house h old appliances 
Alexander Yates, Oren Etzioni, Daniel Weld 

January 2003 Proceedings of the 8th international conference on Intelligent user 
interfaces IUI '03 

Publisher: ACM Press 

Full text available* f 5 "! pdf(233 97 KB) Additional Information: full citation, abstract , referenc es, citings, index 
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As household appliances grow in complexity and sophistication, they become harder and 
harder to use, particularly because of their tiny display screens and limited keyboards. 
This paper describes a strategy for building natural language interfaces to appliances that 
circumvents these problems. Our approach leverages decades of research on planning 
and natural language interfaces to databases by reducing the appliance problem to the 
database problem; the reduction provably maintains desirable pro ... 

Keywords: appliance, database, natural language interface, planner 
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^ P. Mulhem, L. Nigay 
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Research and development in information retrieval SIGIR '96 
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8 ISIS: an adaptive, trilingual conversational system with interleaving interaction and 
dele g ation dialo gs 

Helen Meng, P. C. Ching, Shuk Fong Chan, Yee Fong Wong, Cheong Chat Chan 
September 2004 ACM Transactions on Computer-Human Interaction (TOCHI), volume n 

Issue 3 

Publisher: ACM Press 

Full text available: ^|pdf(3.71 MB) Additional Information: fu ll citat ion, a bstract , references, index terms 

ISIS (Intelligent Speech for Information Systems) is a trilingual spoken dialog system 
(SDS) for the stocks domain. It handles two dialects of Chinese (Cantonese and 
Putonghua) as well as English— the predominant languages in our region. The system 
supports spoken language queries regarding stock market information and simulated 
personal portfolios. The conversational interface is augmented with a screen display that 
can capture mouse-clicks as well as textual input by typing or stylus- writing. ... 

Keywords: Human-computer spoken language interface, interaction and delegation 
dialogs 




9 Military applications: human systems modeling II: Explorin g the constraints of human Q 
be havior represen tation 

John C. Giordano, Paul F. Reynolds, David C. Brogan 

December 2004 Proceedings of the 36th conference on Winter simulation WSC '04 
Publisher: Winter Simulation Conference 

Full text available: ^.Rlf(322._39KB) Additional Information: fujl .citation, abstract, references 

Human behavior representation (HBR) is an elusive, yet critical goal for many in the 
simulation community. Requirement specifications related to HBR often exceed current 
capabilities. There exist a number of tools, techniques and frameworks to model and 
simulate HBR, but they are constrained and do not generalize well. Even with a vibrant 
research community, certain HBR characteristics remain beyond our grasp, unless some 
unforeseen disruptive technologies emerge. We survey the state of the prac ... 



10 E-ma rketin g & e-businesses: Desi gning intelli g ent sales-a g ent for o n line selli ng 





Shiu-li Huang, Fu-ren Lin 

August 2005 Proceedings of the 7th international conference on Electronic commerce 



ICEC 05 

Publisher: ACM Press 

Full text available: ^ pdf( 523.25 KB) Additional Information: full citatio n, abstract , references , index terms 

Online purchase from e-stores is getting popular as the prevalence of electronic 
commerce. At current stage, most e-stores resemble vending machines rather than real 
stores because they lack clerks to persuade prospects into buying products and to bargain 
with the customers for making a good deal. This research designs an easy-to-use and 
autonomous sales-agent to act as a virtual clerk in an e-store, and then investigates 
whether an e-store with this virtual clerk could increase customers' produ ... 

Keywords: abstract argumentation framework, negotiation, persuasion, reinforcement 
learning, sales-agent 
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interfaces IUI '06 

Publisher: ACM Press 

Full text available* 1?!| odf(427 81 KB) Additional Information: full., c , itati on t abstract, r eferenc es, dteiby, index 
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We are reaching a crisis with design of user interfaces for consumer electronics. Flashing 
12:00 time indicators, push-and-hold buttons, and interminable modes and menus are all 
symptoms of trying to maintain a one-to-one correspondence between functions and 
physical controls, which becomes hopeless as the number of capabilities of devices grows. 
We propose instead to orient interfaces around the goals that users have for the use of 
devices. We present Roadie, a user interface agent that provides ... 

Keywords: commonsense reasoning, consumer electronics, goal-oriented interfaces, 
planning 
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Jon Doyle, Thomas Dean 

December 1996 ACM Computing Surveys (CSUR), volume 28 issue 4 
Publisher: ACM Press 

Full text available: ^|pdf( 243.02 KB ) Additional Information: full cita tion, referenc es, index terms 



13 An intellig ent distr i buted environment for active learnin g 
Yi Shang, Hongchi Shi, Su-Shing Chen 

April 2001 Proceedings of the 10th international conference on World Wide Web 
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Publisher: ACM Press 

Full text available: f£lpdf( 200.31 KB ) Additional Information: full citation , references , citings, index terms 
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14 Session 5C: conversational agen ts: A plug-in architecture for generating collaborative 
<£> ag ent responses 

^ Charles Rich, Neal Lesh, Andrew Garland, Jeff Rickel 

July 2002 Proceedings of the first international joint conference on Autonomous 
agents and multiagent systems: part 2 AAMAS '02 

Publisher: ACM Press 

Full text available* I 5 ?! pdf(453 82 KB) Additional Information: full citation , abstract , re ferences , citin gs, index 
• |£| : terms 

We describe an implemented architecture for programming the responses of collaborative 
interface agents out of easily composable and reusable plug-in components, and discuss 
the underlying theoretical and practical issues. The power of the architecture comes 
primarily from a rich representation of collaborative discourse state, which includes a 
focus stack and plan tree. The architecture also provides a useful separation between the 
principles and preferences underlying an agent's behavior. We i ... 

Keywords: action selection and planning, agent architectures, conversational agents, 
interface agents 
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Charles Rich 

January 1996 ACM SIGCHI Bulletin, Volume 28 issue l 
Publisher: ACM Press 

Full text available: ^| pdf(1 .03 MB ) Additional Information: full citation, a bstract , dtings, index terms 

An implemented system is described which allows a software agent to collaborate with a 
human user using a shared application window. The system automatically controls input 
permission and also provides mechanisms for signalling and communication. A 
generalization of the system to multiple users and agents, called NShare, is compared 
with common window-sharing tools, such as SharedX. This work is part of a larger agenda 
to apply principles of human collaboration and discourse structure to human-c ... 

16 Perce ptual user interfaces: multimodal interfaces that process what comes naturall y 
Sharon Oviatt, Philip Cohen 

March 2000 Communications of the ACM, volume 43 issue 3 
Publisher: ACM Press 
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17 Confirmation in multimodal systems 

David R. McGee, Philip R. Cohen, Sharon Oviatt 

August 1998 Proceedings of the 36th annual meeting on Association for 

Computational Linguistics - Volume 2 , Proceedings of the 17th 
international conference on Computational linguistics - Volume 2 

Publisher: Association for Computational Linguistics 

Full text available: ^| pdf(1.12 MB) Additional Information: f ull citatio n, ab stract , refer ences , citin gs 

Systems that attempt to understand natural human input make mistakes, even humans. 
However, humans avoid misunderstandings by confirming doubtful input. Multimodal 
systems— those that combine simultaneous input from more than one modality, for 
example speech and gesture-have historically been designed so that they either request 
confirmation of speech, their primary modality, or not at all. Instead, we experimented 
with delaying confirmation until after the speech and gesture were combin ... 

Keywords: confirmation, disambiguation, multimodal, uncertainty 
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Publisher: ACM Press 
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Most computational spoken dialogue systems take a "literary" approach to reference 
resolution. With this type of approach, entities that are mentioned by a human interactor 
are unified with elements in the world state based on the same principles that guide the 
process during text interpretation. In human-to-human interaction, however, referring is 
a much more collaborative process. Participants often under-specify their referents, 
relying on their. discourse partners for feedback if more informa ... 

Keywords: HCI, dialogue systems, eye tracking, reference resolution 
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Laurence Nigay, Joelle Coutaz 

May 1993 Proceedings of the SIGCHI conference on Human factors in computing 
systems CHI '93 

Publisher: ACM Press 

Full text available: -ffl odf( 806.88 KB ) Additional Information: full citation , abstract, references , citings, index 

terms 

Multimodal interaction enables the user to employ different modalities such as voice, 
gesture and typing for communicating with a computer. This paper presents an analysis of 
the integration of multiple communication modalities within an interactive system. To do 
so, a software engineering perspective is adopted. First, the notion of "multimodal 
system" is clarified. We aim at proving that two main features of a multimodal system are 
the concurrency of processing and the fusion ... 

Keywords: concurrency, data fusion, design space, modality, multimodal interaction, 
software architecture, taxonomy 
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